Random ideas

Evolution as a basis for AGI

Idea for dealing with overfitting; consider “good” weights for two mostly disjoint yet representative subsets of samples from the same distribution, we would like the weights on one subset to be very similar to the weights of the other subset as an indicator that the weights are indeed representative. Find a way to enforce this and balance accuracy with this similarity.

Transformer architecture: do we need all these encoder/decoder attention layers? Perhaps experiment without them.

Transformer construction: Q, K, V. Queries, keys, values are generated separately for each position. In other words, functional relationships between them are indirectly learned. Perhaps we could cut the query and key from some same cloth (i.e. intermediate embedding) to better enforce a “similarity” relationship?

Keyboard optimization with RL based on statistical letter correlations and evaluated on aggregate proximity/distance (with constraints in keyboard shape and size)

Program synthesis through RL where reward function is test cases passed

Deep learning: formalization of “network capacity/power” for a given architecture and learning of that and good capacities for given learning tasks

RL with learning of “policy learning mechanism” I.e. dynamics between reward and how to move network weights according to that. Look into RL algos and updates

ChatGPT involving research papers

My own search engine for the internet. Google sucks.

Gradient descent, but using another neural net to help with updates. Currently we add the gradient to the parameters, which gets less and less meaningful as we get further away from our parameter point. Perhaps try to tease out or approximate loss shape instead of always applying addition.

NLP with knowledge (graphs, ontologies, etc.) and unlabelled data; perhaps leverage and distill existing large models trained on vast data; text mining application; intelligence defined on target; concepts

Evolving neural networks; genes are both mutating and their space is extremely flexible in terms of biological outcome; neural networks are also very flexible learners (universal approximators); look into gene mutations and simple DNA/genetics vs. complex and human genes

Humans evolved to communicate pass down information and knowledge obtained through intelligence; this accumulated intelligence over large swaths of time and overcomes the problem of death; ideas, knowledge, skills, infrastructure survive though individuals die, enabling the human species to be effective